Write pickle to file-like without intermediate in-memory buffer #37056

ig248 · 2020-10-11T16:15:57Z

Before this change, calling pickle.dumps() created an in-memory byte buffer, negating the advantage
of zero-copy pickle protocol 5. After this change, pickle.dump writes directly to open file(-like),
cutting peak memory in half in most cases.

Profiling was done with pandas@master and python 3.8.5

Related issues:

ENH: Support out-of-band pickling (protocol 5) #34244

Update: ASV results

$ asv continuous -f 1.1 origin/master HEAD -b pickle yields:

       before           after         ratio
     [58f74686]       [3c1ee270]
                      <zero-copy-pickle>
-           91.9M            80.1M     0.87  io.pickle.Pickle.peakmem_write_pickle

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

closes ENH: Support out-of-band pickling (protocol 5) #34244
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Before this change, calling pickle.dumps() created an in-memory byte buffer, negating the advantage of zero-copy pickle protocol 5. After this change, pickle.dump writes directly to open file(-like), cutting peak memory in half in most cases.

jreback

nice, can you add a memory asv and show the results of this & a whatsnew note for 1.2 perf section.

jreback · 2020-10-11T16:25:03Z

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

ig248 · 2020-10-11T16:30:03Z

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default?

jreback · 2020-10-11T16:33:57Z

also to actually close this issue i think would be good to add some tests that explicity set the pickle protocal to 5

Isn't that covered by py38 tests where pickle.HIGHEST_PROTOCOL (==5) is used by default?

probably but let's make it explict as well (e.g. parameterize on 4,5,highest when PY38 or greater)

ig248 · 2020-10-11T17:31:36Z

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

jreback · 2020-10-11T17:44:10Z

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

it should show a diff here in any event. what kind of results are you seeing?

pandas/tests/io/test_pickle.py

ig248 · 2020-10-11T18:13:30Z

Added ASV peakmem_ benchmark. The benchmark currently uses an ~11MB dataframe - I have no experience with ASV to tell how reliable memory benchmarking is for small-ish objects, any advice appreciated.

it should show a diff here in any event. what kind of results are you seeing?

Yep, added results above - peakmem_write_pickle shows the 11MB drop nicely - in practice the relative gain is obv much more significant than shown for larger dataframes.

pep8speaks · 2020-10-11T18:31:48Z

Hello @ig248! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-10-12 22:51:15 UTC

jreback

lgtm ping on green. can you post the results of the asv (in the top of the PR, just edit it)

doc/source/whatsnew/v1.2.0.rst

pandas/tests/io/test_pickle.py

jreback · 2020-10-12T22:51:37Z

lgtm @ig248 just resolved the conflict and can merge on green.

jreback · 2020-10-14T12:27:44Z

thanks @ig248 very nice!

…as-dev#37056)

ig248 changed the title ~~[Draft] Write pickle to file-like without intermediate storage~~ Write pickle to file-like without intermediate in-memory buffer Oct 11, 2020

jreback requested changes Oct 11, 2020

View reviewed changes

jreback added IO Pickle read_pickle, to_pickle Performance Memory or execution speed performance labels Oct 11, 2020

ig248 added 3 commits October 11, 2020 17:52

Added note in whatsnew for v1.2.0

e357724

Add explicit tests for default and specific pickling protocols

82b046f

Add asv peakmem benchmark for to_pickle/from_pickle

3c1ee27

jreback requested changes Oct 11, 2020

View reviewed changes

pandas/tests/io/test_pickle.py Outdated Show resolved Hide resolved

pandas/tests/io/test_pickle.py Outdated Show resolved Hide resolved

Clean up test parametrization over supported protocols

f35c59d

ig248 added 2 commits October 11, 2020 19:34

Fix linter errors

7db5697

isort imports

cccfb6e

jreback requested changes Oct 12, 2020

View reviewed changes

doc/source/whatsnew/v1.2.0.rst Outdated Show resolved Hide resolved

pandas/tests/io/test_pickle.py Show resolved Hide resolved

jreback added this to the 1.2 milestone Oct 12, 2020

ig248 and others added 2 commits October 12, 2020 18:23

Format whatsnew entry

19557b4

Merge branch 'master' into zero-copy-pickle

497fa70

jreback approved these changes Oct 12, 2020

View reviewed changes

jreback merged commit 0fa47b6 into pandas-dev:master Oct 14, 2020

JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Oct 26, 2020

Write pickle to file-like without intermediate in-memory buffer (pand…

439746e

…as-dev#37056)

kesmit13 pushed a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020

Write pickle to file-like without intermediate in-memory buffer (pand…

8fe3956

…as-dev#37056)

asishm mentioned this pull request Jan 10, 2021

BUG:to_pickle() raises TypeError when compressing large dataframe #39002

Closed

3 tasks

twoertwein mentioned this pull request Jan 24, 2021

REGR: write compressed pickle files with protocol=5 #39376

Merged

4 tasks

jakirkham mentioned this pull request Oct 21, 2022

PERF: Improve pickle support with BZ2 & LZMA #49068

Merged

5 tasks

jakirkham mentioned this pull request Nov 6, 2022

Use latest pickle dask/distributed#5826

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write pickle to file-like without intermediate in-memory buffer #37056

Write pickle to file-like without intermediate in-memory buffer #37056

ig248 commented Oct 11, 2020 •

edited

Loading

jreback left a comment

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020 •

edited

Loading

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020

pep8speaks commented Oct 11, 2020 •

edited

Loading

jreback left a comment

jreback commented Oct 12, 2020

jreback commented Oct 14, 2020

Write pickle to file-like without intermediate in-memory buffer #37056

Write pickle to file-like without intermediate in-memory buffer #37056

Conversation

ig248 commented Oct 11, 2020 • edited Loading

Update: ASV results

jreback left a comment

Choose a reason for hiding this comment

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020 • edited Loading

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020

jreback commented Oct 11, 2020

ig248 commented Oct 11, 2020

pep8speaks commented Oct 11, 2020 • edited Loading

Comment last updated at 2020-10-12 22:51:15 UTC

jreback left a comment

Choose a reason for hiding this comment

jreback commented Oct 12, 2020

jreback commented Oct 14, 2020

ig248 commented Oct 11, 2020 •

edited

Loading

ig248 commented Oct 11, 2020 •

edited

Loading

pep8speaks commented Oct 11, 2020 •

edited

Loading